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(54) Motion compensated video signal processing 

(57) In a motion compensated video signal processing system, each correlation surface is combined with a 
group of correlation surfaces from surrounding search blocks (each correlation surface representing the 
correlation between a search block of one image with a search area in a subsequent image) to generate a 
plurality of combined or grown correlation surfaces and a motion vector is generated from each of the original 
and combined correlation surfaces. 

Each correlation surface is subjected to an array of delays, delays 650 being equal to the transmission 
time for one correlation surface, whilst the row delays 660 correspond to the transmission time of a row of 
correlation surfaces. The outputs of each of the delays in the array are therefore identical points in the 
surfaces. The outputs from selected delays are normalised and combined (Figure 16) to generate, as 
described, five combined correlation surfaces (Figures 4B - 4F), form each of which a motion vector is 
generated. 
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At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy. 

The date of filing shown above is that provisionally accorded to the application in accordance with the provisions of 
Section 15(4) of the Patents Act 1977 and is subject to ratification or amendment. 
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MOTION COMPENSATED VIDEO SIGNAL PROCESSING 



This invention relates to motion compensated video signal processing. 
Motion compensated video signal processing is used in applications such as 
5 television standards conversion, film standards conversion and conversion between 

video and film standards. 

In a motion compensated television standards convener, such as the converter 
described in UK patent application number GB-A-2 231 749, pairs of successive input 
images are processed to generate sets of motion vectors representing image motion 

10 between the pair of input images. The processing is carried out on discrete blocks 

of the images, so that each motion vector represents the inter- image motion of the 
content of a respective block. 

In the motion vector estimation process a correlation surface representing 
spatial correlation between blocks of the two input images is examined to detect 

15 points of maximum correlation. (The correlation surface actually represents the 
difference between the two input images so that the points of maximum correlation 
are in fact minima on the correlation surface). When a minimum has been detected, 
a motion vector is generated from the spatial position of the minimum in the 
correlation surface. Tests are performed to determine whether that minimum 

20 represents a significant peak in correlation with respect to the remainder of the 
correlation surface. If the minimum passes this test it is treated as being "valid" , and 
a confidence flag associated with the motion vector is set. 

After the individual motion vectors, termed "local" motion vectors, have been 
derived for the blocks of the input image, the local motion vectors are supplied to a 

25 motion vector reducer. The motion vector reducer assigns to each block a set of 

motion vectors selected, in order of preference, from: the zero motion vector; the 
local motion vector for that block; the local motion vectors for blocks neighbouring 
that block in the input image; and a set of vectors known as "global" motion vectors. 
The global motion vectors are derived for each image by ranking all the (valid) local 

30 motion vectors for this image in order of decreasing frequency, and then selecting 

from the most common of the local motion vectors a number of unique motion 
vectors to be used as global motion vectors. A constraint may be applied here to 



ensure that the global motion vectors differ from each other by at least a minimum 
amount. 

The sets of motion vectors for all the blocks of the input image are then 
supplied to a motion vector selector. The purpose of the motion vector selector is to 
allocate to each picture element (pixel) of a block of an output image a single motion 
vector selected from the set of motion vectors supplied for a corresponding block of 
the input image. As this motion vector will be used in interpolation of the output 
pixel, the motion vectors supplied to the motion vector selector are tested so that the 
"best" motion vector is selected for each pixel from the set of motion vectors for the 
corresponding block. The selected motion vector for each pixel is then supplied to 
a motion compensated interpolator which uses these motion vectors to interpolate the 
pixels of an output image from the pair of input images in dependence upon the 
temporal offset of the output image from the pair of input images. 

If a search block used in the generation of a correlation surface is dominated 
by a large object, the motion of that large object will cause a large minimum in the 
correlation surface. This in turn will mean that a motion vector is generated 
representing the motion of the large object. Because only one motion vector is 
derived from each correlation surface, the motion of a smaller object in the same 
block will be masked by that of the large object. If the smaller object does not 
extend into other blocks, it is likely that a suitable motion vector representing the 
motion of the smaller object will not be generated. 

An example of this problem is shown in Figure 1 of the accompanying 
drawings, in which a large stationary object 10 (e.g. a wall) dominates a search block 
20, which also contains a much smaller moving object 30 (a lorry). A cross-section 
of a correlation surface generated using this search block is shown in Figure 2 of the 
accompanying drawings; a large minimum 40 (from which a motion vector is 
generated) represents the motion of the stationary wall 10, while a smaller, localised, 
minimum 50 (which is ignored in favour of the larger minimum 40) represents the 
lorry's motion. 

This invention provides a motion compensated video signal processing 
apparatus in which motion vectors are generated for use in interpolation of an output 
image of an output video signal, the motion vectors representing image motion 



between a pair of input images of an input video signal, the apparatus comprising: 
means for generating a plurality of correlation surfaces, each representing correlation 
between a respective block of one input image of the pair and an area, comprising a 
plurality of blocks, in the other image of the pair; means for examining the 
correlation surfaces to detect points of maximum correlation within the correlation 
surfaces; and means for generating motion vectors from the detected points of 
maximum correlation in the correlation surfaces; in which, for a correlation surface 
having a plurality of points of local maximum correlation, the means for generating 
motion vectors is operable to generate a corresponding plurality of motion vectors 
from that correlation surface. 

In an apparatus according to the invention, more than one motion vector may 
be generated from a single correlation surface. This means that motion vectors can 
be generated to reflect the motion of a number of objects within a search block, 
rather than simply the largest object or part of the background. 

For convenient later processing of the motion vectors, it is preferable to limit 
the maximum number of motion vectors which can be generated from a single 
correlation surface. To this end, it is preferred that the means for generating motion 
vectors is operable to generate up to a predetermined number of motion vectors from 
each correlation surface. 

Although the predetermined number of motion vectors could be selected in an 
arbitrary fashion in the case of a correlation surface having greater than the 
predetermined number of local correlation maxima, it is preferred that the means for 
generating motion vectors is operable to generate a respective motion vector from the 
predetermined number of most significant local correlation maxima. 

In order to reduce the risk of generating a motion vector, which may 
ultimately be used in the interpolation of a pixel of the output image, from noise or 
alias components in the correlation surfaces, it is preferred that the apparatus 
comprises means for performing a confidence test on each motion vector generated 
from a secondary correlation maximum, a secondary correlation maximum being a 
point of local maximum correlation in a correlation surface other than the point of 
absolute maximum correlation in that correlation surface; and means for preventing 
the use, in interpolation of the output image, of motion vectors derived from 
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secondary correlation maxima and which failed the confidence test. 

In one preferred embodiment the means for performing a confidence test 
comprises: means for detecting whether each motion vector generated from a 
secondary correlation maximum lies within a predetermined threshold amount of one 
or more motion vectors derived from correlation surfaces generated from adjacent 
blocks of the input image. A motion vector generated from a secondary correlation 
maximum passes the confidence test if it does indeed lie within the predetermined 
threshold amount of a neighbouring motion vector. 

Preferably the apparatus comprises means for deriving a set of global motion 
vectors comprising a plurality of distinct motion vectors selected from the most 
common of the motion vectors. 

In another preferred embodiment the means for performing a confidence test 
comprises means for detecting whether each motion vector generated from a 
secondary correlation maximum lies within a predetermined threshold amount of a 
global motion vector. 

In a further preferred embodiment the means for performing a confidence test 
comprises means for detecting a change in sign of the rate of change of a correlation 
surface in a predetermined number of non-parallel linear directions passing through 
a secondary correlation maximum. 

Preferably the predetermined number of non parallel linear directions 
comprises a direction indicative of horizontal image motion and a direction indicative 
of vertical image motion. 

A further criterion may be added to the "rate of change" test described above; 
it is preferred that the means for performing a confidence test comprises means for 
detecting whether the rate of change of the correlation surface adjacent to a secondary 
correlation maximum exceeds a predetermined amount. 

Vector reduction (the assignment of a group of motion vectors to each block, 
from which vectors are later selected for use in interpolation of pixels of the output 
image) is preferably modified to reflect the availability of multiple vectors from a 
single correlation surface. To this end, it is preferred that the apparatus comprises 
means for assigning to each of the blocks a group of motion vectors selected from, 
in the following order of preference: 
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(i) a motion vector indicative of zero image motion; 

(ii) the motion vector generated from a point of absolute maximum 
correlation in the correlation surface derived from that block; 

(iii) motion vectors generated from secondary correlation maxima in the 
5 correlation surface derived from that block; and 

(iv) the global motion vectors. 

Viewed from a second aspect this invention provides a method of motion 
compensated video signal processing, in which motion vectors are generated to 
represent image motion between a pair of input images of an input video signal, the 

10 method comprising the steps of: generating a plurality of correlation surfaces, each 

representing correlation between a respective block of one input image of the pair and 
an area, comprising a plurality of blocks, in the other image of the pair; examining 
the correlation surfaces to detect points of maximum correlation within the correlation 
surfaces; and generating motion vectors from the detected points of maximum 

15 correlation in the correlation surfaces; in which, for a correlation surface having a 
plurality of points of local maximum correlation, the step of generating motion 
vectors comprises generating a corresponding plurality of motion vectors from that 
correlation surface. 

A previously proposed technique for use in a motion compensated standards 
20 conversion apparatus was to "grow" correlation surfaces during vector estimation. 

In this technique, each correlation surface was examined for minima in its own right, 
and was also added to adjacent correlation surfaces to form "grown" surfaces for 
examination. The process of correlation surface "growing" is illustrated in 

Figures 3 and 4a to 4f of the accompanying drawings. Figure 3 shows an array of 
25 search blocks 100 in an input image forming one of a pair of input images from 

which motion vectors are to be derived. A correlation surface is generated from each 
of the search blocks 100 by comparing that search block with a larger search area in 
the other of the pair of input images. Each of a group of 15 search blocks (in fact 
a rectangular array of five search blocks horizontally by three search blocks 
30 vertically) is referred to by a lower case letter from w a" to "o". 

In the previously proposed technique of correlation surface growing t each 
correlation surface is added to predetermined groups of correlation surfaces generated 



from surrounding search blocks. A normalisation process then takes place. An 
example is shown in Figures 4a to 4f, in which six "grown" correlation surfaces 
generated from a correlation surface "h" (Figure 3) are shown. In each case, Figures 
4a to 4f show the spatial relationship of the correlation surfaces from which each 
"grown" correlation surface is generated. In particular, Figure 4a illustrates the 
correlation surface "h" itself; Figure 4b represents a sum of the correlation surfaces 
"g", "h" and "i"; Figure 4b represents a sum of the correlation surfaces "g", "h" and 
"i"; Figure 4c represents a sum of the correlation surfaces "f", "g", "h", "i" and "j"; 
Figure 4d represents a sum of the correlation surfaces "c", "h" and "m"; Figure 4e 
represents a sum of the correlation surfaces "b", "c", "d", "g", "h", "i", "1" "m" 
and "n"; and Figure 4f represents a sum of all of the fifteen correlation surfaces "a" 
to V. 

Figure 5 of the accompanying drawings is a schematic diagram of a previously 
proposed circuit for performing correlation surface growing. Each correlation surface 
is passed sequentially to "growing circuits" 110, which add groups of adjacent 
surfaces to generate the six output surfaces shown in Figures 4a to 4f. Minima 
detection and vector testing (e.g. the confidence test mentioned above) are then 
performed 120 in parallel on each of the six surfaces. The results of the minima 
testing are then passed to selection logic 130 which selects a motion vector derived 
from a single one of the six surfaces, to be passed to vector reduction. 

If a minimum derived from a "grown" surface was used, this meant that vector 
estimation effectively took place over a wider image area. However, the ability to 
track the movement of small objects was reduced. The selection of a motion vector 
derived from one of the "grown" correlation surfaces, to be passed to vector 
reduction, involved a compromise between the detection of detailed morion (by using 
a vector derived from a small correlation surface) and the greater accuracy offered 
by the use of a larger ("grown") correlation surface. 

Viewed from a third aspect this invention provides a motion compensated 
video signal processing apparatus in which motion vectors are generated to represent 
image motion between a pair of input images of an input video signal, the apparatus 
comprising: means for generating a plurality of original correlation surfaces, each 
representing correlation between a respective search block of one input image of the 
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pair and a search area, comprising a plurality of blocks, in the other image of the 
pair; means for combining each original correlation surface with groups of original 
correlation surfaces generated from surrounding search blocks, to generate a 
respective plurality of combined correlation surfaces; and means for generating a 
5 motion vector from each of the original correlation surfaces and the combined 

correlation surfaces. 

According to this aspect of the invention, motion vectors are generated from 
the various "grown" correlation surfaces and are passed to vector reduction. This 
avoids the need to make an early compromise between the detection of detailed 
10 motion (by using a vector derived from a small correlation surface) and the greater 

accuracy offered by the use of a larger ("grown") correlation surface. 

Although various configurations could be employed, it is preferred that the 
means for combining is operable to combine each original correlation surface with 
five groups of original correlation surfaces generated from search blocks in a 
15 surrounding array of search blocks. 

Preferably the array is a 5 x 3 array of search blocks. 
In an advantageously simple embodiment the means for combining comprises: 
means for receiving serial data representing successive original correlation surfaces; 
a plurality of delay circuits operable to generate respective delayed data from the 
20 serial data; and means for adding the delayed data generated by a predetermined 
group of the delay circuits, to generate output data representing a combined 
correlation surface. 

Preferably the means for adding comprises means for performing normalised 
addition of the delayed data. 
25 In a preferred embodiment the apparatus comprises means for deriving a set 

of global motion vectors comprising a plurality of distinct motion vectors selected 
from the most common of the motion vectors. 

Preferably the vector reduction process is modified to reflect the availability 
of multiple motion vectors generated from "grown" correlation surfaces. It is 
30 preferred that the apparatus comprises means for assigning to each of the blocks a 
group of motion vectors selected from, in the following order of preference: 
(i) a motion vector indicative of zero image motion; 
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(ii) the motion vector generated from a selected combined correlation 
surface; 

(iii) motion vectors having at least a predetermined number of occurrences 
amongst those motion vectors generated from the original correlation surfaces derived 
from the current block and adjacent blocks; and 

(iv) the global motion vectors. 

In another embodiment, the motion vectors derived from all of the combined 
correlation surfaces could be selected in preference to the global motion vectors. 

In a preferred embodiment the selected combined correlation surface is a 
combined correlation surface generated by combining original correlation surfaces 
generated from each of the array of search blocks. 

Preferably the apparatus comprises means for examining the correlation 
surface to detect points of maximum correlation within the original and combined 
correlation surfaces; and means for generating motion vectors from the detected 
points of maximum correlation; in which, for a correlation surface having a plurality 
of points of local maximum correlation, the means for generating motion vectors is 
operable to generate a corresponding plurality of motion vectors from that correlation 
surface. It is also preferred that the apparatus comprises means for performing a 
predetermined confidence test on each motion vector; and means for preventing the 
assignment of motion vectors which failed the confidence test. 

Viewed from a fourth aspect this invention provides a method of motion 
compensated video signal processing, in which motion vectors are generated to 
represent image motion between a pair of input images of an input video signal, the 
method comprising the steps of: generating a plurality of original correlation surfaces, 
each representing correlation between a respective search block of one input image 
of the pair and a search area, comprising a plurality of blocks, in the other image of 
the pair; combining each original correlation surface with groups of original 
correlation surfaces generated from surrounding search blocks, to generate a 
respective plurality of combined correlation surfaces; and generating a motion vector 
from each of the original correlation surfaces and the combined correlation surfaces. 

As mentioned above, the purpose of the motion vector reducer is to provide 
the motion vector selector with a reasonable selection of motion vectors for each 



block from which to select the most appropriate motion vector for each output pixel 
corresponding to that block. The global motion vectors each represent a motion 
which was frequently detected in the image as a whole, so that it is often reasonable 
to include these vectors in the sets of vectors from which the most appropriate vectors 
are selected by the motion vector selector. 

However, the motion vector selection process is not completely reliable, and 
may under some circumstances, erroneously select a global motion vector (derived 
from the motion of a different part of the image) rather than the correct local motion 
vector. This can result in disturbing artifacts in the output image, such as those 
illustrated in Figure 6 of the accompanying drawings. Here, a global motion vector 
corresponding to the motion of a vehicle 140 provides a good match between test 
blocks 150, 160 in a stationary window in the background of the image. The vehicle 
motion vector has therefore been inappropriately selected for use in interpolation of 
the region of the image at the centre of the window. The result is a gap 170 in the 
central vertical frame member of the window. 

Viewed from a fifth aspect this invention provides a motion compensated video 
signal processing apparatus in which motion vectors are generated to represent image 
motion between a pair of input images of an input video signal, the apparatus 
comprising: means for generating a plurality of correlation surfaces, each representing 
correlation between a respective block of one input image of the pair and an area, 
comprising a plurality of blocks, in the other image of the pair; means for generating 
a plurality of motion vectors from the correlation surfaces, each motion vector 
depending on a point of maximum correlation in that correlation surface; means for 
deriving a set of global motion vectors comprising a plurality of distinct motion 
vectors selected from the most common of the plurality of motion vectors; means for 
examining the correlation surfaces to detect those global motion vectors which are 
locally valid for each block, by detecting whether the correlation surface generated 
from that block has a local correlation maximum indicative of image motion 
substantially identical to that represented by each global motion vector; and means for 
assigning to each of the blocks a group of motion vectors selected from: the zero 
motion vector; the motion vector representing image motion of that block; and those 
global motion vectors which are locally valid for that block. 
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In accordance with this aspect of the present invention, a test is performed to 
detect whether each global motion vector is "locally valid" for each block of the 
output image, by examining the corresponding correlation surface to detect evidence 
of the motion represented by the global motion vector. If no such evidence is found, 
5 that global vector is not allowed to be selected for possible use in interpolation of the 

output block. This can avoid or alleviate the problems described above. 

Preferably the detecting means comprises: means for detecting a change in 
sign of the rate of change of a correlation surface in a predetermined number of non- 
parallel linear directions passing through a position indicative of image motion 
10 represented by a global motion vector. 

Although various linear directions could be used, in an advantageously simple 
embodiment the predetermined number of non parallel linear directions comprises a 
direction indicative of horizontal image motion and a direction indicative of vertical 
image motion. 

15 Preferably the apparatus comprises means for examining the correlation 

surface to detect points of maximum correlation within the correlation surfaces; and 
in which, for a correlation surface having a plurality of points of local maximum 
correlation, the means for generating motion vectors is operable to generate a 
corresponding plurality of motion vectors from that correlation surface. 

20 In a preferred embodiment the apparatus comprises means for combining each 

correlation surface with groups of correlation surfaces generated from surrounding 
blocks, to generate a respective plurality of combined correlation surfaces; and means 
for generating a respective motion vector from each of the combined correlation 
surfaces. 

25 Viewed from a sixth aspect this invention provides a method of motion 

compensated video signal processing, in which motion vectors are generated to 
represent image motion between a pair of input images of an input video signal, the 
method comprising the steps of: generating a plurality of correlation surfaces, each 
representing correlation between a respective block of one input image of the pair and 

30 an area, comprising a plurality of blocks, in the other image of the pair; generating 
a plurality of motion vectors from the correlation surfaces, each motion vector 
depending on a point of maximum correlation in that correlation surface; deriving a 
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set of global motion vectors comprising a plurality of distinct motion vectors selected 
from the most common of the plurality of motion vectors; detecting those global 
motion vectors which are locally valid for each block, by detecting whether the 
correlation surface generated from that block has a local correlation maximum 
5 indicative of image motion substantially identical to that represented by each global 

motion vector; and assigning to each of the blocks a group of motion vectors selected 
from: the zero motion vector; the motion vector representing image motion of that 
block; and those global motion vectors which are locally valid for that block. 

Apparatus according to the invention may be particularly advantageously 
10 employed in a television standards conversion apparatus. 

Features of the respective aspects of the invention defined above may be 
combined according to all of the possible various combinations. 

An embodiment of the invention will now be described, by way of example 
only, with reference to the accompanying drawings, throughout which like parts are 
15 referred to by like references, and in which: 

Figure 1 is a schematic diagram of a portion of an image representing a large 
stationary object and a small moving object; 

Figure 2 is a schematic diagram illustrating a cross-section through a 
correlation surface generated from the image of Figure 1; 
20 Figure 3 illustrates an array of search blocks; 

Figures 4a to 4f illustrate the "growing" of correlation surfaces; 

Figure 5 is a schematic diagram of a circuit for performing correlation surface 
growing; 

Figure 6 illustrates artifacts in an output image due to the erroneous selection 
25 of a global motion vector; 

Figure 7 is a schematic block diagram of a motion compensated television 
standards converter; 

Figure 8 illustrates vertical subsampling of an interlaced video field; 

Figure 9 is a schematic diagram of a correlation surface; 
30 Figure 10 illustrates a part of a modified motion processor; 

Figure 1 1 is a schematic diagram of a correlation surface having two minima; 

Figure 12 schematically illustrates the examination of a correlation surface 
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along two perpendicular directions; 

Figures 13a and 13b illustrate cross-sections through the correlation surface 
of Figure 12 along the two perpendicular directions; 

Figure 14 illustrates a part of a modified motion processor; 

Figure 15 is a schematic block diagram of a part of a correlation surface 
processor; 

Figure 16 illustrates a normalising adder; 
Figure 17 illustrates a part of a modified motion processor; 
Figure 18 is a representation of the image of Figure 6, showing blocks of the 
image for which evidence of a global vector was found; 
Figures 19 illustrates a correlation surface; and 

Figure 20 illustrates a cross-section through the correlation surface of Figure 

19. 

Figure 7 is a schematic block diagram of a motion compensated television 
standards conversion apparatus. The apparatus receives an input interlaced digital 
video signal 250 (e.g. an 1125/60 2:1 high definition video signal (HDVS)) and 
generates an output interlaced digital video signal 260 (e.g a 1250/50 2:1 signal). 

The input video signal 250 is first supplied to an input buffer/packer 310. In 
the case of a conventional definition input signal, the input buffer/packer 310 formats 
the image data into a high definition (16:9 aspect ratio) format, padding with black 
pixels where necessary. For a HDVS input the input buffer/packer 310 merely 
provides buffering of the data. 

The data are passed from the input buffer/packer 310 to a matrix circuit 320 
in which (if necessary) the input video signal's colorimetry is converted to the 
colorimetry of the desired output signal, such as the standard "CCIR recommendation 
601" (Y,Cr,Cb) colorimetry. 

From the matrix circuit 320 the input video signal is passed to a time base 
changer and delay 330, and via a sub-sampler 370 to a subsampled time base changer 
and delay 380. The time base changer and delay 330 determines the temporal 
position of each field of the output video signal, and selects the two fields of the input 
video signal which are temporally closest to that output field for use in interpolating 
that output field. For each field of the output video signal, the two input fields 



13 

selected by the time base changer are appropriately delayed before being supplied to 
an interpolator 340 in which that output field is interpolated. A control signal t, 
indicating the temporal position of each output field with respect to the two selected 
input fields, is supplied from the time base changer and delay 330 to the interpolator 
340. 

The subsampled time base changer and delay 380 operates in a similar 
manner, but using spatially subsampled video supplied by the subsampler 370. Pairs 
of fields, corresponding to the pairs selected by the time base changer 330, are 
selected by the subsampled time base changer and delay 380 from the subsampled 
video, to be used in the generation of motion vectors. 

The time base changers 330 and 380 can operate according to synchronisation 
signals associated with the input video signal, the output video signal, or both. In the 
case in which only one synchronisation signal is supplied, the timing of fields of the 
other of the two video signals is generated deterministically within the time base 
changers 330, 380. 

The pairs of fields of the subsampled input video signal selected by the 
subsampled time base changer and delay 380 are supplied to a motion processor 385 
comprising a direct block matcher 390, a correlation surface processor 400, a motion 
vector estimator 410, a motion vector reducer 420, a motion vector selector 430 and 
a motion vector post-processor 440. The pairs of input fields are supplied first to the 
direct block matcher 390 which calculates correlation surfaces representing the spatial 
correlation between search blocks in the temporally earlier of the two selected input 
fields and (larger) search areas in the temporally later of the two input fields. 

From the correlation surfaces output by the block matcher 390, the correlation 
surface processor 400 generates a larger number of interpolated correlation surfaces, 
which are then passed to the motion vector estimator 410. The motion vector 
estimator 410 detects points of greatest correlation in the interpolated correlation 
surfaces. (The original correlation surfaces actually represent the difference between 
blocks of the two input fields; this means that the points of maximum correlation are 
in fact minima on the correlation surfaces, and are referred to as "minima"). In order 
to detect a minimum, additional points on the correlation surfaces are interpolated, 
providing a degree of compensation for the loss of resolution caused by the use of 
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subsampled video to generate the surfaces. From the detected minimum on each 
correlation surface, the motion vector estimator 410 generates a motion vector which 
is supplied to the motion vector reducer 420. 

The motion vector estimator 410 also performs a confidence test on each 
5 generated motion vector to establish whether that motion vector is significant above 

the average data level, and associates a confidence flag with each motion vector 
indicative of the result of the confidence test. The confidence test, known as the 
"threshold" test, is described (along with certain other features of the apparatus of 
Figure 7) in GB-A-2 231 749. 

10 A test is also performed by the motion vector estimator 410 to detect whether 

each vector is aliased. In this test, the correlation surface (apart from an exclusion 
zone around the detected minimum) is examined to detect the next lowest minimum. 
If this second minimum does not lie at the edge of the exclusion zone, the motion 
vector derived from the original minimum is flagged as being potentially aliased. 

15 The motion vector reducer 420 operates to reduce the choice of possible 

motion vectors for each pixel of the output field, before the motion vectors are 
supplied to the motion vector selector 430. The output field is notionally divided into 
blocks of pixels, each block having a corresponding position in the output field to that 
of a search block in the earlier of the selected input fields. The motion vector 

20 reducer compiles a group of four motion vectors to be associated with each block of 

the output field, with each pixel in that block eventually being interpolated using a 
selected one of that group of four motion vectors. 

Vectors which have been flagged as "aliased" are re-qualified during vector 
reduction if they are identical to non-flagged vectors in nearby blocks. 

25 As part of its function, the motion vector reducer 420 counts the frequencies 

of occurrence of "good" motion vectors (i.e. motion vectors which pass the 
confidence test and the alias test, or which were re-qualified as non-aliased), with no 
account taken of the position of the blocks of the input fields used to obtain those 
motion vectors. The good motion vectors are then ranked in order of decreasing 

30 frequency. The most common of the good motion vectors which are significantly 
different to one another are then classed as "global" motion vectors. Three motion 
vectors which pass the confidence test are then selected for each block of output 
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pixels and are supplied, with the zero motion vector, to the motion vector selector 
430 for further processing. These three selected motion vectors are selected in a 
predetermined order of preference from: 

(i) the motion vector generated from the corresponding search block (the 
5 "local" motion vector"); 

(ii) those generated from surrounding search blocks ("neighbouring" 
motion vectors); and 

(iii) the global motion vectors. 

The motion vector selector 430 also receives as inputs the two input fields 
10 which were selected by the subsampled time base changer and delay 380 and which 
were used to calculate the motion vectors. These fields are suitably delayed so that 
they are supplied to the motion vector selector 430 at the same time as the vectors 
derived from them. The motion vector selector 430 supplies an output comprising 
one motion vector per pixel of the output field. This motion vector is selected from 
15 the four motion vectors for that block supplied by the motion vector reducer 420. 

The vector selection process involves detecting the degree of correlation 
between test blocks of the two input fields pointed to by a motion vector under test. 
The motion vector having the greatest degree of correlation between the test blocks 
is selected for use in interpolation of the output pixel. A "motion flag" is also 
20 generated by the vector selector. This flag is set to "static" (no motion) if the degree 
of correlation between blocks pointed to by the zero motion vector is greater than a 
preset threshold. 

The vector post-processor reformats the motion vectors selected by the motion 
vector selector 430 to reflect any vertical or horizontal scaling of the picture, and 

25 supplies the reformatted vectors to the interpolator 340. Using the motion vectors, 

the interpolator 340 interpolates an output field from the corresponding two (non- 
subsampled) interlaced input fields selected by the time base changer and delay 330, 
taking into account any image motion indicated by the motion vectors currently 
supplied to the interpolator 340. 

30 If the motion flag indicates that the current output pixel lies in a moving or 

temporally changing part of the image, pixels from the two selected fields supplied 
to the interpolator are combined in relative proportions depending on the temporal 
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position of the output field with respect to the two input fields (as indicated by the 
control signal t), so that a larger proportion of the nearer input field is used. If the 
motion flag is set to "static" then the temporal weighting is fixed at 50% of each 
input field. The output of the interpolator 340 is passed to an output buffer 350 for 
5 output as a high definition output signal, and to a down-converter 360 which 

generates a conventional definition output signal 365, using the motion flag to control 
filter selection. 

The down-converter 360 allows a representation of the output of the apparatus 
(which may be, for example, a high definition video signal) to be monitored, 

10 transmitted and/or recorded using conventional definition apparatus. This has benefits 

because conventional definition recording equipment is significantly cheaper and very 
much more widespread than high definition equipment. For example, a simultaneous 
output of conventional and high definition video may be required for respective 
transmission by terrestrial and satellite channels. 

15 The subsampler 370 performs horizontal and vertical spatial subsampling of 

the input video fields received from the matrix 320, before those input fields are 
supplied to the time base changer 380. Horizontal subsampling is a straightforward 
operation in that the input fields are first prefiltered by a half-bandwidth low pass 
filter (in the present case of 2:1 horizontal decimation) and alternate video samples 

20 along each video line are then discarded, thereby reducing by one half the number of 

samples along each video line. 

Vertical subsampling of the input fields is complicated by the fact that, in this 
embodiment, the input video signal 250 is interlaced. This means that successive 
lines of video samples in each interlaced field are effectively two video lines apart, 

25 and that the lines in each field are vertically displaced from those in the preceding or 

following field by one video line of the complete frame. 

One approach to vertical subsampling would be to perform progressive scan 
conversion (to generate successive progressively scanned video frames each having 
1 125 lines) and then to subsample the progressively scanned frames by a factor of 2 

30 to perform the vertical subsampling. However, efficient progressive scan conversion 

would require a degree of motion compensated processing, which processing could 
adversely affect the operation of the motion processor 385. Furthermore, real-time 
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progressive scan conversion of a high definition video signal would require 
particularly powerful and complex processing apparatus. 

A simpler approach to vertical spatial subsampling is shown in Figure 8, in 
which the input fields are first low pass filtered in the vertical direction (to reduce 
potential aliasing) and a filtering operation is then performed which effectively 
displaces each pixel vertically by half a video line downwards (for even fields) or 
upwards (for odd fields). The resulting displaced fields are broadly equivalent to 
progressively scanned frames which have been subsampled vertically by a factor of 
two. 

In summary, therefore, the result of the subsampling operations described 
above is that the motion processor 385 operates on pairs of input fields which are 
spatially subsampled by a factor of two in the horizontal and the vertical directions. 
This reduces the processing required for motion vector estimation by a factor of four. 

Figure 9 is a schematic diagram of a correlation surface 500. The correlation 
surface represents the difference between a search block of the earlier of the two 
input fields from which the surface is generated, and a (larger) search area in the later 
of the two input fields. A peak in correlation is therefore represented by a minimum 
point 510 on the correlation surface 500. The position of the minimum point 510 on 
the correlation surface 500 determines the magnitude and direction of the motion 
vector derived from that correlation surface. 

As mentioned above, if a search block used for the generation of a correlation 
surface is dominated by a large object (which may be moving or stationary), then the 
detected motion of that large object will cause a correspondingly large minimum in 
the correlation surface. If one motion vector is generated from each correlation 
surface, the motion vector will reflect the motion of the large object, thus masking 
the motion of any smaller objects in the same block. This means that if a small 
moving object falls within a single search block, then in an apparatus in which only 
one motion vector is generated from each correlation surface, there is no way of 
accurately representing the motion of the small object. 

In accordance with one embodiment of the present invention, Figure 10 shows 
a modification of part of the motion processor 385* in which a plurality of motion 
vectors may be generated from each interpolated correlation surface output by the 
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correlation surface processor 400. In particular, a modified vector estimator 550 and 
a modified vector reducer 560 are employed. 

Before the operation of the vector estimator 550 will be described, the nature 
of a correlation surface having multiple minima representing the motion of objects of 
5 differing sizes within the search block will first be described. An example of such 

a correlation surface is illustrated schematically in Figure 11. 

In Figure 11 a correlation surface 570 has a large minimum 580 and a 
secondary minimum 590. The minimum 580 represents the point of minimum 
difference (maximum correlation) on the entire correlation surface. However, in 

10 contrast, the minimum 590 is only a secondary (localised) minimum representing a 

point of greater correlation than the immediately surrounding parts of the correlation 
surface. Accordingly, the vector estimator 550 must detect localised as well as 
absolute minima in the correlation surface. 

In operation, the vector estimator 550 generates one motion vector for each 

15 minimum detected in each interpolated correlation surface, up to a predetermined 

maximum number of minima. In the example shown in Figure 10, up to six motion 
vectors are generated from each interpolated correlation surface and are passed to the 
vector reducer 560. If the correlation surface has more than six minima, motion 
vectors are generated from the six most significant minima. 

20 It is desirable to take precautions against certain problems which can arise 

when multiple minima are detected from a single correlation surface. In the example 
shown in Figure 1 1 , the two minima reflect the motions of a large and a small object 
within the same search block. However, in an image having an area of regularly 
spaced patterning, multiple minima can be detected which are regularly spaced across 

25 the correlation surface. One of these minima will represent the correct motion of the 

image patterning; the others are due to alias interactions between the image patterning 
and the size of the search blocks used to generate the correlation surfaces. In order 
to help prevent the use of an incorrect motion vector in the interpolation of the output 
image, the multiple motion vectors generated from a single correlation surface are 

30 subjected to thorough testing before use. Four possible techniques for testing a 

plurality of motion vectors derived from a single correlation surface are: 
a) Each of the motion vectors derived from secondary minima may be compared 



19 

with vectors derived from neighbouring correlation surfaces. This comparison, which 
may be performed in the vector reducer 560, allows a motion vector derived from a 
secondary minimum to be used for interpolation of the output image if that motion 
vector lies within a predetermined threshold of a motion vector derived from a 
5 neighbouring correlation surface. 

b) A motion vector derived from a secondary minimum may only be used if that 
vector is substantially identical to a known global motion vector. The comparison of 
secondary minima and global motion vectors will be described in greater detail below. 

c) A differential technique may be employed, in which the differential (rate of 
10 change) of the correlation surface is examined along various linear directions. A 

change in sign of the differential indicates a localised minimum in the direction of 
examination. If there is a change in sign of the differential along a predetermined 
number of directions, that minimum can be used to generate a motion vector. 

An example of technique (c) is illustrated in Figures 12, 13a and 13b. In 

15 Figure 12, a correlation surface 600 is examined along two perpendicular directions 

A-A and B-B. Figure 13a illustrates a cross-section through the correlation surface 
along the direction A-A and Figure 13b illustrates a cross-section through the 
correlation surface along the direction B-B. 

The differential (rate of change) of the correlation surface is evaluated along 

20 the two directions A-A and B-B. As shown in Figures 13a and 13b, where the 
differential changes sign (indicating a localised minimum) along both of the two 
perpendicular directions A-A and B-B, an allowable localised minimum is indicated 
and a motion vector is generated from that localised minimum. 

d) As a modification of technique (c) motion vectors are generated from localised 
25 minima only if the differential either side of the localised minimum is greater than a 

threshold value. 

Each of the motion vectors derived from a single correlation surface are 
passed by the vector estimator 550 to the vector reducer 560. The vector reducer 560 
then selects four motion vectors to be assigned to each block of the output image, in 
30 a similar manner to the vector reducer 420 described above. The four motion vectors 

for each block are selected from (in the following order of preference): 

(i) the zero motion vector; 
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(ii) the local motion vector derived from the most significant minimum in 
the correction surface for that block; 

(iii) local motion vectors derived from secondary minima in that correlation 
surface; and 

5 (iv) global motion vectors. 

Figure 14 is a schematic block diagram of a modified motion processor 385" 
according to a second embodiment of the invention. A modified correlation surface 
processor 610 effects 'growing' of the interpolated correlation surfaces by combining 
each interpolated correlation surface with groups of other interpolated correlation 
10 surfaces in a predetermined spatial relationship to that correlation surface. Various 

ones of the groups of correlation surfaces are illustrated in Figures 4a to 4f as 
described above. 

Each of the six 'grown' correlation surfaces output by the correlation surface 
processor is passed to the vector estimator 620. In contrast to the previously 
15 proposed apparatus illustrated in Figure 5, the vector estimator 620 generates a 

motion vector from each of the grown correlation surfaces and passes those motion 
vectors to a vector reducer 630. 

The vector reducer receives the motion vectors derived from the various 
grown correlation surfaces and selects four motion vectors for each block to be passed 
20 to vector selection. These four motion vectors are selected from (in the following 
order of preference): 

(i) the zero motion vector; 

(ii) the motion vector generated from a selected combined correlation 
surface (e.g. the grown correlation surface of Figure 4f); 

25 (iii) motion vectors having at least a predetermined number of occurrences 

amongst those motion vectors generated from the original correlation surfaces derived 
from the current block and adjacent blocks; and 

(iv) the global motion vectors. 

In another embodiment, the motion vectors derived from all of the combined 
30 correlation surfaces could be selected in preference to the global motion vectors. 

The motion vectors are tested using the confidence test described above. Only 
those motion vectors which passed the confidence test are selected to be supplied to 



21 

vector selection. 

Figure 15 is a schematic block diagram of the part of the correlation surface 
processor 610 in which the 'growing* of correlation surfaces is carried out. 

In the apparatus of Figure 15, input data 640 representing the interpolated 
correlation surfaces are supplied in serial form to a number of correlation surface 
delays 650. Each correlation surface delay 650 delays the input data 640 by a period 
equivalent to the transmission time of the data representing one correlation surface. 
This means that the data at the input and the output of each of the correlation surface 
delays 650 represent identical points within two adjacent correlation surfaces. 

The input data 640 are also supplied to two row delays 660, 670, which delay 
the data by a period equivalent to the transmission time of a complete row of 
correlations surfaces. This means that the data at the input and the output of a row 
delay (660 or 670) represent identical points within two correlation surfaces at 
corresponding positions in two adjacent rows. The output of each of the row delays 
660, 670 is supplied to a series of four correlation surface delays 650. 

The input and output of each correlation surface delay 650 is annotated by a 
lower case letter from a to o. The data at these points represent respective parts of 
a 5 x 3 array of correlation surfaces in the form shown in Figure 3. Accordingly, 
in order to generate one of the grown correlation surfaces illustrated in Figures 4a to 
4f, the data and the appropriate outputs from the apparatus of Figure 15 is added and 
normalised. 

An example of the use of a normalising adder 680 is shown schematically in 
Figure 16, in which the grown correlation surface illustrated in Figure 4d is 
generated. In particular, the data at the outputs c, h and m of the apparatus of Figure 
15 are supplied to the normalising adder 680. The normalising adder 680 generates 
output data according to the following function: 

CSout = (c+h+m)/3 

The correlation surface in Figure 4a (which is simply the current correlation 
surface) does not require a normalising adder of the type shown in Figure 16. 
However, normalising adders similar to that shown in Figure 16 are used to generate 
the remaining grown correlation surfaces of Figures 4b to 4f. 

Figure 17 is a schematic block diagram of part of the motion processor 385"* 
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according to a third embodiment of the invention. In this embodiment, interpolated 
correlation surfaces are supplied from the correlation surface processor 400 in parallel 
to the vector estimator 410 and to a modified vector reducer 700. Vector estimation 
is carried out in the manner described above with reference to Figure 7. 
5 In the motion vector reducer 700, each correlation surface is inspected at the 

position of the global vectors. A differential test (of the type described above with 
reference to Figures 12, 13a and 13b) is performed to detect whether a localised 
minimum exists at that position. If such a localised minimum exists, then that global 
vector is allowed to be used for the current block of the output image. 

10 The effect of this modified vector reduction process is that a global vector is 

only passed to vector selection for a particular block if there is some evidence that 
the global vector represents the motion of at least a small part of that block. 

Figure 18 is a representation of the image shown in Figure 6, showing those 
blocks 710 of the image for which evidence of a global vector representing the motion 

15 of the car was found. Use of that global vector is restricted to the blocks 710, which 

helps to avoid the erroneous use of the global vector described with reference to 
Figure 6. 

Figures 19 and 20 illustrate the test performed by the vector reducer 700 to 
examine the correlation surface for evidence of each global vector. Figure 19 

20 illustrates a correlation surface 720 on which the predominant minimum 
(corresponding to the local vector) has coordinates (V l(x) , V l(y) ). At the position of a 
global motion vector, namely (V g(x) , V^), the correlation surface 720 is examined 
along two perpendicular directions to detect evidence of a localised minimum. 

Figure 20 is a cross-section through the correlation surface 720 passing 

25 through the predominant minimum at (V l(Jt) , V I(y) ) and the position of the global 

vector, namely (V g(x) , V g(y) ). In this case, a localised minimum is detected at the 
position of the global vector, and the global vector is passed for possible use in 
interpolation of pixels of the current block of the output image. 

If desired, the apparatus of Figures 10, 14 and 17 may be combined in various 

30 combinations so that, for example, the vector estimator derives one or more motion 
vectors from each grown correlation surface, with correlation surface re-inspection 
being also employed. 
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CLAIMS 

1. Motion compensated video signal processing apparatus in which motion 
vectors are generated to represent image motion between a pair of input images of an 

5 input video signal, the apparatus comprising: 

means for generating a plurality of original correlation surfaces, each 
representing correlation between a respective search block of one input image of the 
pair and a search area, comprising a plurality of blocks, in the other image of the 
pair; 

10 means for combining each original correlation surface with groups of original 

correlation surfaces generated from surrounding search blocks, to generate a 
respective plurality of combined correlation surfaces; and 

means for generating a motion vector from each of the original correlation 
surfaces and the combined correlation surfaces. 

15 

2. Apparatus according to claim 1 , in which the means for combining is operable 
to combine each original correlation surface with five groups of original correlation 
surfaces generated from search blocks in a surrounding array of search blocks. 

20 3. Apparatus according to claim 2, in which the array is a 5 x 3 array of search 

blocks. 

4. Apparatus according to any one of claims 1 to 3, in which the means for 
combining comprises: 

25 means for receiving serial data representing successive original correlation 

surfaces; 

a plurality of delay circuits operable to generate respective delayed data from 
the serial data; and 

means for adding the delayed data generated by a predetermined group of the 
30 delay circuits, to generate output data representing a combined correlation surface. 



Apparatus according to claim 4, in which the means for adding comprises 
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means for performing normalised addition of the delayed data. 

6. Apparatus according to any one of claims 1 to 5, comprising means for 
deriving a set of global motion vectors comprising a plurality of distinct motion 

5 vectors selected from the most common of the motion vectors. 

7 . Apparatus according to claim 6, comprising means for assigning to each of the 
blocks a group of motion vectors selected from, in the following order of preference: 

(i) a motion vector indicative of zero image motion; 
10 (ii) the motion vector generated from a selected combined correlation 

surface; 

(iii) motion vectors having at least a predetermined number of occurrences 
amongst those motion vectors generated from the original correlation surfaces derived 
from the current block and adjacent blocks; and 
15 (iv) the global motion vectors. 

8. Apparatus according to claim 2 and claim 7, in which the selected combined 
correlation surface is a combined correlation surface generated by combining original 
correlation surfaces generated from each of the array of search blocks. 

20 

9. Apparatus according to any one of claims 1 to 8, comprising: 

means for examining the correlation surface to detect points of maximum 
correlation within the original and combined correlation surfaces; and 

means for generating motion vectors from the detected points of maximum 
25 correlation; 

in which, for a correlation surface having a plurality of points of local 
maximum correlation, the means for generating motion vectors is operable to generate 
a corresponding plurality of motion vectors from that correlation surface. 

30 10. Apparatus according to claim 9, comprising: 

means for performing a confidence test on each motion vector; and 

means for preventing the assignment of motion vectors which failed the 
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confidence test. 

11. A method of motion compensated video signal processing, in which motion 
vectors are generated to represent image motion between a pair of input images of an 
5 input video signal, the method comprising the steps of: 

generating a plurality of original correlation surfaces, each representing 
correlation between a respective search block of one input image of the pair and a 
search area, comprising a plurality of blocks, in the other image of the pair; 

combining each original correlation surface with groups of original correlation 
10 surfaces generated from surrounding search blocks, to generate a respective plurality 

of combined correlation surfaces; and 

generating a motion vector from each of the original correlation surfaces and 
the combined correlation surfaces. 

15 12. Television standards conversion apparatus comprising apparatus according to 

any one of claims 1 to 10. 

13. Motion compensated video signal processing apparatus substantially as 
hereinbefore described with reference to the accompanying Figures 14, 15 and 16. 

20 

14. A method of motion compensated video signal processing, the method being 
substantially as hereinbefore described with reference to the accompanying Figures 
14, 15 and 16. 

25 15. Television standards conversion apparatus substantially as hereinbefore 

described with reference to the accompanying Figures 14, 15 and 16. 
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